Method of modifying facial images, makeup simulation method, makeup method, makeup support apparatus and foundation transfer film

ABSTRACT

New findings such as the highlight theory which determine the impression of a face are utilized for the technique of modifying a photographed facial image and the makeup technique. A method according to the present invention includes determining a desirable face desired by a person who is subjected to facial modification or makeup, and performing image processing like shape-merging on an original facial image or an image of a face of the subject person based on the desirable face to thereby prepare images of selectable modified faces.

TECHNICAL FIELD

The present invention relates to a technique of modifying facial imagesand a makeup art of putting makeup on a face using the modifyingtechnique.

BACKGROUND OF THE INVENTION

The present inventor has been making studies, for a long time, onfactors which determine the faces of persons, particularly facialimpressions, and beauty and ugliness, from the viewpoints of plasticsurgery and cosmetic surgery. The study has provided several newfindings. One of them is the “highlight theory”. Conventionally, it wasa general thought that the components of a face, such as eyebrows, eyes,a nose and lips, would become significant factors to determine a facialimpression. The research done by the present inventor however shows thatthe prime factor to determine a facial impression is the shading stateof a face, i.e., the states of a bright area and a dark area,particularly, the state of the brightest area or highlighted area. Theshading state relates to shading on a face which occurs underillumination of the natural light, and reflects or expresses thethree-dimensional shape of the face. The states of a highlighted area insuch shading particularly affects a facial impression.

The analysis based on this highlight theory classifies faces into threegeneral types. As exemplarily shown in FIG. 5, the shapes of highlightedareas are an inverted triangle type [(a) in FIG. 5], an oval type [(b)in FIG. 5] and a clover-leaf type [(c) in FIG. 5]. A typical beautifulface in each facial type has a good shape of the associated highlightedarea. With regard to an inverted triangle type, for example, no orlittle distortion or deformation on its shape is a significant factor ingiving a good impression on a face.

Another finding is a theory on the shapes of faces people feel beautifulor preferable. Conventional theories on facial beauty include canontheory or a theory based on golden cut, and the average theory which hasrecently come up. The average theory means a theory that the face thatappears most beautiful among those of a certain human race is an averageface of the individual faces of that race. Faces based on those theoriesmerely provide general “beautiful faces”. In other words, those theoriescannot give ideal post-makeup faces which are makeup aims applicable tomakeup methods. Unlike those theories, the theory found by the presentinventor can be put to a practical use in makeup methods.

According to what has been newly found by the present inventor, throughthe following procedures, it is possible in most occasions to find themost beautiful or preferable face for each specific person, or the mostbeautiful or preferable face that can be made by makeup available tothat individual. The fundamental theory is such that if faces areproduced by combining a face of given standards, specifically, a facegenerally considered beautiful or a face preferred by an individual andthe actual face of that particular individual at the proper mixingratios through an image processing technique, an “ideal face” which isconsidered most beautiful or preferable by that individual can be foundin most cases from the foregoing mixed faces. Specifically, one of modelfaces, which are, for example, faces of preferred talents or actressesor actors, or “beautiful faces” originated from the average theory andthe like, is selected as a desirable face. Based on this desirable face,the actual face of the person is subjected to image processing likemorphing so that the actual face is mixed with the desirable face toapproach the latter. This process provides a plurality of “modifiedfaces” with different mixing ratios or similarity levels with respect tothe desirable face. In most cases, those “modified faces” contain an“ideal face” that person considers most beautiful or preferable. Thedata obtained by the present inventor shows that persons in 48 cases outof 50 could find “ideal faces” they thought most beautiful orpreferable. In the other two cases, “ideal faces” could be found bymodifying the desirable faces. The mixing ratio in every case rangedfrom 10 to 50%.

It is to be noted that one model face itself should not necessarily bethe most preferable face for a particular person. That is, the “idealface” a particular person can find for herself or himself is a “modifiedface” which is resulted from the mixing of a model face with her or hisown face, not the model face itself. The actual results of theexperiments conducted by the present inventor show that most subjectpersons did not prefer model faces using “beautiful faces” created bythe average theory or the like.

The present invention has been devised to effectively utilize theabove-described findings and aims at using those findings to a techniqueof modifying, for example, photographed facial images and to a makeuptechnique.

DISCLOSURE OF THE INVENTION

According to the present invention, facial images picked up by a stillcamera, a video camera or the like are modified as follows. First, adesirable face is determined, and then an original facial image issubjected to image processing to become closer to this desirable face sothat a modified facial image is acquired.

Model faces can be used in determining a desirable face. The faces ofpreferable talents or actresses or actors can be used as model faces. Ifone does not have any preferable talent or the like, standardgood-looking faces acquired from a plurality of beautiful faces whichhave previously been prepared corresponding to human races or likeaccording to the foregoing average theory may be used. In the foregoingmethod, it may also be arranged that an original facial image issubjected to image processing to become closer to a desirable face,thereby forming images of a plurality of selectable modified faces withdifferent similarity levels with respect to the desirable face, and thena selection is made from those selectable modified faces to acquire amodified facial image. This further facilitates acquiring a face idealto the subject person.

It is preferable to use the foregoing highlight theory in modifying andmaking an original facial image closer to a desirable face. That is, thekey point is to make the states of highlighted areas of a facial imageto be modified closer to those of a desirable face. This can provide amore beautiful and preferable face and thus facilitate approaching to adesirable face.

This facial image modifying method is an application of the foregoingtheory on the shapes of faces that people think beautiful or preferable.Unlike the conventional photograph modification techniques which mainlydepend on a manual work, this method can automate the modification workby electronic processing. What is more, modification results can givegreater satisfaction. This method of the present invention can beadapted to operational simulation in plastic surgery and cosmeticsurgery as well as ordinary photograph modification.

In a makeup simulation method embodying the present invention and amakeup method which uses this simulation method, a desirable face isdetermined first. In the case where a makeup instructor instructs asubject person how to put makeup, for example, a desirable face may bedetermined by asking the subject person about preferences.Alternatively, model faces may be used in determining a desirable face.The faces of talents, or actresses or actors the subject person like canbe used as model faces. If the subject person has no preference for anytalent or the like, averages of good-looking faces which have previouslybeen prepared corresponding to human races or the like may be used asmodel faces. In the former case, a preferred desirable face can beselected from model faces using the name or the like of the preferabletalent or the like as an index. In the latter case, a preferreddesirable face should be selected from model faces of standard beautywhich has previously been input as data. In both cases a selection ismade while sequentially displaying the model faces on a monitor screenor based on the type of the face of the subject person. In the lattercase, the model faces should not necessarily be shown to the subjectperson.

Once the desirable face is decided, then images of imagined post-makeupfaces based on the desirable face or post-makeup faces which areimagined as preferable are prepared. This involves a process of mixingthe face of the subject person with the desirable face to make theformer face closer to the latter by performing image processing likeshape-merging on the image of the face of the subject person. Then, anideal post-makeup face the subject person desires is determined from theimagined post-makeup faces. Specifically, a plurality of imaginedpost-makeup faces with different mixing ratios or similarity levels withrespect to the desirable face are acquired through the foregoing imageprocessing, and a preferable selection is made from those faces as anideal post-makeup face within the range of the similarity levels whichcan be obtained by makeup. Thus, the ideal post-makeup face which isexpected to be finally obtained is given in advance. That is, thesubject person can know the makeup-finished face in a short time. Asapparent from the above, the significant feature of the makeup methodembodying the present invention lies in that a final, makeup-finishedface can be presented in a short time by determining a desirable face onwhich a makeup procedure is based and acquiring an ideal post-makeupface with respect to this desirable face. In this feature is effectivelyused the above-described theory on the shapes of faces people thinkbeautiful or preferable.

When an ideal post-makeup face is decided through the foregoing makeupsimulation, a makeup procedure is derived from this ideal post-makeupface. Specifically, a sequence of makeup procedures for achieving theideal post-makeup face, such as selecting which portions of eyebrows tobe shaved and where to draw eyebrows, selecting the lines and ranges foreye liners and eye shadows, selecting the colors of eye shadows, settingthe drawing range for a lipstick and separately applying differentfoundations, is derived from a previously set makeup program. Based onthis makeup procedure, makeup is applied to the subject person. Thismakes it possible to accurately realize the ideal post-makeup face orthe makeup state the subject person has previously found appropriate, onthe face of the subject person. That is, makeup which the subject persondesires can be done freely and in a short time.

To realize the foregoing feature of the makeup method of the presentinvention which lies in the preparation of an ideal post-makeup facethrough image processing based on a desirable face or a model face, itis important to mix the current face of a subject person with a modelface through image processing to get the former face closer to thelatter. That is, it is essential to acquire an ideal post-makeup face bymaking the current face of the subject person closer to the referencemodel face by image processing. The feature also requires an effectivescheme or standards for making the current face of the subject personcloser to the ideal post-makeup face by actual makeup. This can beaccomplished by the foregoing highlight theory.

Based on the highlight theory which teaches highlighted areasconsiderably affect the impression of a face and a beautiful andpreferable face can be made by erasing any unnecessary shade (dark area)to make the shapes of the highlighted areas better, it becomesrelatively easy to get the face of a subject person closer to a modelface through image processing and it is also easy to make the face ofthe subject person closer to a desirable face by implementing such amakeup as to adjust the states of the highlighted areas in associationwith those of the model face (desirable face). In other words, a moreeffective scheme of making the face of the subject person closer to thereference model face can be acquired based on the highlight theory.

The image processing for preparing the images of imagined post-makeupfaces on the basis of the highlight theory involves a process of makingthe shapes of the eyebrows, the eye lines, the lip line, and so forthcloser to those of a desirable face within a range where modification bymakeup is possible, in addition to the basic process of getting thestates of highlighted areas on a face closer to those of the desirableface within a range where modification by makeup is possible. Ashape-merging scheme or the like can be also used in this case.

Based on the highlight theory, attention should be paid to a greatesthighlighted area as an impressing element of a face in the makeupsimulation method and makeup method, too. Since makeup about highlightedareas is accomplished by separately applying different foundations, thetypes of foundations and procedures for applying them separately arederived as essential factors from an ideal post-makeup face.

To accomplish the process, image processing is performed to divide theideal post-makeup face into a plurality of areas originating fromdifferences between bright and dark areas like a pattern of contourlines. Then, color data is acquired for each segmented area. Color datain this case consists of hue and brightness, and chroma if necessary,and R, G and B which are used in a television system or C, M, Y and Kwhich are basic printing colors in a computer system are normally used.Based on the color data of the individual areas and a previously setfoundation program, foundations to be applied on the individual areasare determined. Finally, the thud determined foundations are applied onthe face of the subject person in accordance with the segmented areas.

One makeup simulation method according to the present invention and amakeup method using this simulation method therefore include thefollowing steps: (a) determining a desirable face and performing imageprocessing on an image of a face of a subject person based on thedesirable face, thereby forming images of imagined post-makeup faces;(b) displaying the imagined post-makeup faces, prepared in the step (a),on a monitor screen or the like and selectively determining an idealpost-makeup face desired by the subject person from the displayedimagined post-makeup faces; (c) deriving a makeup procedure from animage of the ideal post-makeup face, determined in the step (b), basedon a previously set makeup program; and (d) applying makeup on the faceof the subject person in accordance with the makeup procedure derived inthe step (c).

Another makeup simulation method embodying the present invention andanother makeup method using this simulation method, particularly, amakeup simulation method for applying makeup based on dark and brightstates on a face and a makeup method using this simulation method,include the following steps: (e) determining a desirable face andperforming image processing on an image of a face of a subject personbased on the desirable face, thereby forming images of imaginedpost-makeup faces; (f) displaying the imagined post-makeup faces,prepared in the step (e), on a monitor screen or the like andselectively determining an ideal post-makeup face desired by the subjectperson from the displayed imagined post-makeup faces; (g) performingimage processing to segment an image of the ideal post-makeup face,determined in the step (f), to a plurality of areas in regard to brightareas and dark areas; (h) acquiring color data for the segmented areasobtained in the step (g); (i) determining foundations to be applied onindividual areas based on the color data for those areas, acquired inthe step (h), and a previously set foundation program; and (j)separately applying foundations, determined in the step (i), on the faceof the subject person in association with the individual areas.

The makeup method based on dark and bright states may be modified insuch a way as to be able to print the image of the ideal post-makeupface, segmented into a plurality of areas in regard to bright areas anddark areas in the step (g), in an actual size of the face of the subjectperson, to cut out individual areas from the printed image to prepare apattern mask for foundation, and to execute separate application offoundations in the step (k) by using this pattern mask. This design canallow different foundations to be separately applied on the associatedareas easily and accurately. That is, separate application of differentfoundations associated with an ideal post-makeup face can be carried outeasily and accurately.

The makeup method which is based on the highlight theory may be modifiedin such a manner as to be able to project the image of the idealpost-makeup face, segmented into a plurality of areas in regard tobright areas and dark areas in the step (g), in an actual size on theface of the subject person, and to execute separate application offoundations in the step (j) based on this projected image. Thismodification can also permit different foundations to be separatelyapplied on the associated areas easily and accurately.

Another makeup method according to the present invention which is basedon dark and bright states includes the following steps: (A) determininga desirable face and performing image processing on an image of a faceof a subject person based on the desirable face, thereby forming imagesof imagined post-makeup faces; (B) displaying the imagined post-makeupfaces, prepared in the step (A), on a monitor screen or the like andselectively determining from the displayed imagined post-makeup faces anideal post-makeup face the subject person desires; (C) performing imageprocessing to segment an image of the ideal post-makeup face, determinedin the step (B), to a plurality of areas in regard to bright areas anddark areas; (D) acquiring color data for the segmented areas obtained inthe step (C); (E) determining foundations to be applied on individualareas based on the color data for those areas, acquired in the step (D),and a previously set foundation program; (F) printing foundations,determined in the step (E) in association with the plurality of areassegmented in regard to bright areas and dark areas in the imageprocessing in the step (C), on a thin base material to prepare atransfer film; and (G) transferring foundations, printed on the transferfilm prepared in the step (F), on the face of the subject person tothereby apply foundations, associated with the individual areas, on theface of the subject person.

A further makeup method according to the present invention which isbased on dark and bright states includes the following steps: (H)determining a desirable face and performing image processing on an imageof a face of a subject person based on the desirable face, therebyforming images of imagined post-makeup faces; (I) displaying theimagined post-makeup faces, prepared in the step (H), on a monitorscreen or the like and selectively determining from the displayedimagined post-makeup faces an ideal post-makeup face the subject persondesires; (K) printing an image of the ideal post-makeup face in thebright and dark states, determined in the step (I), on a thin basematerial using foundations to prepare a foundation transfer film; and(L) transferring foundations, printed on the foundation transfer filmprepared in the step (K), on the face of the subject person to therebyapply the foundations on the face of the subject person.

Those methods can repeatedly realize makeup for an ideal post-makeupface by carrying out simple procedures, such as transferring foundationsfrom a transfer film accurately corresponding to foundations for theideal post-makeup face and properly shading off the transferredfoundations at the boundaries between the individual areas. Thus,desired makeup can be carried out more easily and in a shorter time.

A makeup support apparatus according to the present invention for use inthe foregoing makeup simulation methods and makeup methods comprisesimage storage means for storing images of a plurality of model faces;imaging and input means for imaging and inputting a face of a subjectperson; a monitor for displaying a necessary image; and processing meanscapable of performing a process of causing an image of the face of thesubject person to approach a desirable face already determined tothereby form images of imagined post-makeup faces, a process ofdisplaying the imagined post-makeup faces on a screen of the monitor,and a process of deriving a makeup procedure from a determined idealpost-makeup face.

It is preferable that the processing means in the makeup supportapparatus may further perform a process of segmenting the determinedideal post-makeup face to a plurality of areas in regard to bright areasand dark areas, a process of acquiring color data for the segmentedareas, and a process of determining foundations to be applied onindividual areas based on the acquired color data for those areas.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for explaining one example of image processing in afacial image modifying method.

FIG. 2 is a structural diagram of a makeup support apparatus accordingto one carrying-out mode of the present invention.

FIG. 3 is an explanatory diagram concerning a highlight theory.

FIG. 4 is a diagram for explaining segmentation into highlighted areas.

FIG. 5 is an explanatory diagram of the types of highlighted areas.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinbelow, carrying-out modes of the present invention will described.To begin with, a carrying-out mode of a facial image modifying methodaccording to the present invention will be described. Normally, a systemwhich includes a data processing apparatus like a personal computer anda monitor connected to the apparatus is used in implementing the facialimage modifying method of the present invention. An image of a face tobe modified or an original facial image is put into the data processingapparatus. Then, a desirable face is selected from previously preparedmodel faces, e.g., multiple standard faces which have been preparedbased on the average theory, and the original facial image is subjectedto image processing based on this desirable face.

The image processing is executed by a program which employs an imageprocessing scheme like morphing or twining. Specifically, the coordinatevalues of facial constituents, such as eyebrows, eyes, a nose and amouth, of both an original facial image M and a desirable face T, andthe coordinate values of bright and dark patterns of each face areacquired as shown in FIG. 1, for example, and the original facial imageM is mixed with and transformed toward the desirable face T based onthose coordinate values.

More specifically, first, essential portions, such as eyebrows, eyes, anose, a mouth and the lines of highlighted areas, are plotted for eachof the original facial image M and the desirable face T. For example, atip portion of the nose is expressed as “0”, the eye lines as “1-2” and“3-4”, and the eyebrows as “5-6” and “7-8”. Then, those points on theoriginal facial image M are shifted toward the corresponding points onthe desirable face T by a morphing program. The lines of the highlightedareas or the like are shifted or their inclinations are changed by atwining program. Such image processing by shifting points or the likeincludes a plurality of phases according to the degree of such shifting.That is, selectable modified faces (imagined post-makeup faces in thecase of a makeup method which will be described later) corresponding tothose phases can be obtained. If point shifting is performed every onetenth of the distance between corresponding points on both facialimages, for example, there are ten phases and there are thus tenimagined post-makeup faces. According to the experiments conducted bythe present inventor, most of the times, subject persons found their“ideal faces” (selectable modified faces or imagined post-makeup faces)on selectable modified faces at the second or third phase, i.e.,selectable modified faces which had approached to their desirable facesby degrees of 20 to 30%. That is, while there is a desirable face as animage, a face too close to this desirable face comes off the actual“ideal face”.

As the above process forms the images of a plurality of selectablemodified faces with different similarity levels with respect to adesirable face, they are displayed on the screen of the monitor topermit the subject person to make the best selection, which is treatedas a modified facial image.

Now, carrying-out modes of a makeup simulation method and a makeupmethod according to the present invention will be described. The makeupmethod of the present invention is carried out by using a makeup supportapparatus. The makeup support apparatus of which structure isillustrated in FIG. 2 includes a data processing apparatus 1, a storagedevice 2, a monitor 3 and an image sensing unit 4. The data processingapparatus 1, which executes various kinds of data processing required inthe makeup method of the present invention, is a personal computer orthe like. The storage device 2 is used to store data of model faceswhich are used in the makeup method of the present invention. Aninternal memory device or an external memory device is used as thestorage device 2.

In an exemplified case where a makeup instructor instructs a subjectperson a makeup procedure using such a makeup support system, the makeupmethod proceeds as follows. First, the instructor asks the subjectperson specific questions to determine the impression of a face desiredby the subject person or a desirable face. Model faces are used to makea desirable face concrete. As model faces are used image data of facesof popular talents, actresses or actors or the like, or standardgood-looking faces prepared in association with human races or the like,which are stored in the storage device 2. A desirable face is selectedfrom the model faces by using an average face for each race, each era,each personality, each work or the like or the name of a specifictalent, or is selected from various model faces sequentially displayedon the screen of the monitor 3.

Once the desirable face is determined, the face of the subject person ispicked up by the image sensing unit 4 and its image is sent into thestorage device 2. By performing image processing on the image of theface of the subject person using the data processing apparatus 1, theimage of the face of the subject person and the image of the desirableface are mixed to make the former image closer to the latter. In theimage processing, the original facial image of the face of the subjectperson acquired in the above process is subjected to the same scheme asemployed in the above-described modifying method to make the highlightedareas, the shapes of the eyebrows, the eye lines, the lip line and thelike on the face approach those of the desirable face step by step.

FIG. 3 exemplifies highlighted areas. (a) in FIG. 3 shows the states ofbright and dark areas on the original facial image of the face of thesubject person, and (b) in FIG. 3 shows the states of bright and darkareas made close to those of the desirable face in image processing. Inthe figure, the densely dotted portions represent relatively darkportions or dark areas and a portion surrounded by the dark portionsrepresent the brightest area (highlighted area). As apparent from thisexample in which the highlighted area belongs to an oval shape, thecurrent face of the subject person is off the ideal oval shape and theface of the subject person can be made closer to the desirable facewhich has an ideal oval highlighted area by mainly modifying the offportions.

As this process proceeds, the images of a plurality of imaginedpost-makeup faces with different similarity levels with respect to adesirable face are formed as described above, and those images aredisplayed on the screen of the monitor 3 to permit the subject person tomake a selection. If the subject person agrees with any of the displayedimages, the selected face is set as an ideal post-makeup face.

Once the ideal post-makeup face is determined, a sequence of makeupprocedures about portions of eyebrows to be shaved and where to draweyebrows, the lines and ranges for eye liners and eye shadows, thedrawing range for a lipstick and separate application of differentfoundations, is derived from the ideal post-makeup face. This sequenceis carried out by a previously set makeup program.

With regard to foundation, in particular, how to apply foundations isderived by performing the following processes under the control of afoundation program which is included in the makeup program. First, imageprocessing is performed to divide the ideal post-makeup face into aplurality of bright and dark areas, e.g., two to four areas. Thisprocess is exemplified in FIG. 4. In the exemplified diagram, the faceis segmented to four areas with different brightnesses (black-filledportion, densely-dotted portion, thinly-dotted portion and whiteportion). Next, color data for the segmented areas are acquired. Thecolor data in this case are based on R. G and B used in a televisionsystem. Based on the obtained color data of each area, a foundation tobe applied on that area is determined.

As the makeup procedure needed for making up the face of the subjectperson into the ideal post-makeup face and cosmetics to be used in themakeup procedure are determined in the above manner, actual makeup isfinally applied to the face of the subject person based on what has beendetermined. In utilizing the thus derived makeup procedure and the likein the actual makeup, data about the makeup procedure, etc. should bedisplayed on the monitor screen or printed out.

According to a second carrying-out mode of the present invention, apattern mask is used to apply foundations on the face of a subjectperson. A pattern mask is formed like a pattern sheet by cutting outindividual areas from a printed image of the face, segmented into aplurality of areas in regard to bright areas and dark areas in theabove-described image processing, in the real size of the face of thesubject person. Then, the pattern mask is placed on the face of thesubject person and foundations for the individual areas are applied overthe pattern mask.

A third carrying-out mode of the present invention employs a projectionscheme to apply foundations on the face of a subject person.Specifically, the image of the face, segmented into a plurality of areasin regard to bright areas and dark areas in the above-described imageprocessing, is projected on the face of the subject person in real size,and foundations for the associated areas are separately applied based onthis projected image.

A fourth carrying-out mode of the present invention uses a transferscheme to apply foundations on the face of a subject person.Specifically, first, in association with a plurality of areas segmentedin regard to bright areas and dark areas in the above-discussed imageprocessing, foundations for the respective areas are printed in anactual size on a thin base material to prepare a foundation transferfilm. Alternatively, images of bright and dark states on the idealpost-makeup face determined in the above-described manner are printed ona thin base material using foundations to prepare a foundation transferfilm. Then, such a foundation transfer film is pressed against the faceof the subject person to transfer the foundations. Finally, the propershade-off process or the like is performed on the transferredfoundations at the boundaries between the individual areas. Thefoundation transfer film in this case does not necessarily correspond tothe entire face, but a partial transfer film for the upper eyelids or aportion around the nose, for example, may be used.

According to the fourth carrying-out mode of the present invention,application of foundations to the face of the subject person isaccomplished by directly printing the foundations on the face by meansof a three-dimensional printer or the like. Specifically, an image withseparately applied foundations, which is formed by the above-describedimage processing, is directly spray-printed on the face by athree-dimensional printer or the like which uses the necessaryfoundations as inks.

1-13. (canceled)
 14. A foundation transfer film for use in separateapplication of foundations, said foundation transfer film havingfoundations printed on a thin base material in regard to bright areasand dark areas on a face.
 15. The foundation transfer film of claim 14,wherein the foundation transfer film corresponds to a portion of a face.